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RIGID MULTIVIEW VARIETIES 


MICHAEL JOSWIG, JOE KILEEL, BERND STURMFELS, AND ANDRE WAGNER 


Abstract. The multiview variety from computer vision is generalized 
to images by n cameras of points linked by a distance constraint. The 
resulting five-dimensional variety lives in a product of 2n projective 
planes. We determine defining polynomial equations, and we explore 
generalizations of this variety to scenarios of interest in applications. 


1. Introduction 

The emerging field of Algebraic Vision is concerned with interactions be¬ 
tween computer vision and algebraic geometry. A central role in this en¬ 
deavor is played by projective varieties that arise in multiview geometry [5j. 

The set-up is as follows: A camera is a linear map from the three- 
dimensional projective space to the projective plane both over M. 
We represent n cameras by matrices Ai, A 2 ,..., An G of rank 3. The 
kernel of Aj is the focal point fj G P^. Each image point uj G P^ of camera 
Aj has a line through fj as its fiber in P^. This is the hack-projected line. 

We assume throughout that the focal points of the n cameras are in 
general position, i.e. all distinct, no three on a line, and no four on a plane. 
Let fdjk denote the line in P^ spanned by the focal points fj and fk- This 
is the baseline of the camera pair Aj,Ak. The image of the focal point fj 
in the image plane P^ of the camera Ak is the epipole Ckt^j. Note that the 
baseline fdjk is the back-projected line of with respect to Aj and also 
the back-projected line of with respect to A^. See Figurej^for a sketch. 
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Fix a point X in which is not on the baseline f 3 jk, and let Uj and Uk be 
the images of X under Aj and Ak. Since X is not on the baseline, neither 
image point is the epipole for the other camera. The two back-projected 
lines of Uj and Uk meet in a unique point, which is X. This process of 
reconstructing X from two images Uj and Uk is called triangulation [H § 9 . 1 ]. 

The triangulation procedure amounts to solving the linear equations 




■ X ■ 




(1) 

Bik 

-A, 

= 0 

where B^^ = 

Aj Uj 0 

A]^ 0 U}^ 


For general data we have rank(il'^^) = rank(i?^^) = • • • = rank(i?^^) = 5 , 
where is obtained from by deleting the ith row. Cramer’s Rule 
can be used to recover X. Let AbBf^ G be the column vector formed 
by the signed maximal minors of Bj^. Write A^Bj^ G for the first four 
coordinates of A^Bj^. These are bilinear functions of Uj and u^- They yield 

(2) X = = A5.Bf = ... = . 


We note that, in most practical applications, the data ui,...,Un will be 
noisy, in which case triangulation requires techniques from optimization [T]. 

The multiview variety Va of the camera configuration A = (^i,..., An) 
was defined in [ 3 ] as the closure of the image of the rational map 

, . (/>A : P^ p2 X p2 X • • • X p2, 

X ^ {A,X,A2X,...,AnX). 


The points {ui,U2, ■ ■ ■, Un) G Va are the consistent views in n cameras. The 
prime ideal I a of Va was determined in O Corollary 2 . 7 ]. It is generated by 
the (2) bilinear polynomials det(i?-^^) plus (3) further trilinear polynomials. 
See [8] for the natural generalization of this variety to higher dimensions. 

The analysis in [ 3 j was restricted to a single world point X G P^. In this 
paper we study the case of two world points X,V G P^ that are linked by a 
distance constraint. Consider the hypersurface V(Q) in P^ x P^ defined by 

(4) Q = (X0Y3 - ¥0X3)^ + (XiTa - ¥1X3}^ + (XaTa - >2X3)' - 

The affine variety VjR(Q) C {^3=^3=!} in x consists of pairs of points 
whose Euclidean distance is I. The rigid multiview map is the rational map 
i)A ■ V{Q) ^ p3 X p3 --4 (p2)’" X (P^)*", 

{X,Y) ^ {{AiX,...AnX),iAiY,...AnY)). 

The rigid multiview variety is the image of this map. This is a 5 -dimensional 
subvariety of (P^)^"'. Its multihomogeneous prime ideal Ja lives in the 
polynomial ring M[tt,u] = R[uio,Uii,Ui2, Vio,Vii,Vi2 : i = 1 ,... ,n], where 
{uiQ\Uii\Ui2) and {viQ'.Vii-.Vi2) are coordinates for the fth factor P^ on the 
left respectively right in (P^)*^ x (P^)”. Our aim is to determine the ideal 
Ja- Knowing generators of Ja has the potential of being useful for design¬ 
ing optimization tools as in [T] for triangulation in the presence of distance 
constraints. 

The choice of world and image coordinates for the camera configuration 
A = {Ai, ..., An) gives our problem the following group symmetries. Let N 
be an element of the Euclidean group of motions SE( 3 , M), which is generated 
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by rotations and translations. We may multiply the camera configuration 
on the right by N to obtain AN = {AiN,..., A^N). Then Ja = Jan 
since V{Q) is invariant under SE( 3 ,M). For G GL( 3 ,M), we 

may multiply A on the left to obtain A' = [Mi A ,..., M^A). Then Ja' = 

[Ml® ...® Mn)JA- 

This paper is organized as follows. In Section 2 we present the explicit 
computation of the rigid multiview ideal for n = 2 , 3 , 4 . Our main result, 
to be stated and proved in Section 3 , is a system of equations that cuts 
out the rigid multiview variety V[Ja) for any n. Section 4 is devoted to 
generalizations. The general idea is to replace V[Q) by arbitrary subvarieties 
of that represent polynomial constraints on m > 2 world points. We 

focus on scenarios that are of interest in applications to computer vision. 

Our results in Propositions and Corollary [^ar e proved by compu¬ 

tations with Macaulay 2 [ 3 ]; for details see Appendix | A[ Following standard 
practice in computational algebraic geometry, we carry out the computation 
on many samples in a Zariski dense set of parameters, and then conclude 
that it holds generically. 


2. Two, Three and Four Cameras 


In this section we offer a detailed case study of the rigid multiview variety 
when the number n of cameras is small. We begin with the case n = 2 . The 
prime ideal Ja lives in the polynomial ring M[tt, u] in 12 variables. This is the 
homogeneous coordinate ring of (P^)^, so it is naturally Z^-graded. The vari¬ 
ables uio,uii,ui2 have degree (1,0,0,0), the variables U2o,U2i,U22 have de¬ 
gree (0,1, 0,0), the variables uio, fii, ^12 have degree (0,0,1,0), and the vari¬ 
ables V2o,V2i,V22 have degree ( 0 , 0 , 0 , 1 ). Our ideal Ja is Z^-homogeneous. 

Throughout this section we shall assume that the camera configuration 
A is generic in the sense of algebraic geometry. This means that A lies 
in the complement of a certain (unknown) proper algebraic subvariety in 
the affine space of all n-tuples of 3 x 4 -matrices. All our results in Section 
2 were obtained by symbolic computations with sufficiently many random 
choices of A (see Appendix for details). Such choices of camera matrices 
are generic. They will be attained with with probability 1 . 


Proposition 1. For n = 2 , the rigid multiview ideal Ja is minimally gen¬ 
erated by eleven 7 ,'^-homogeneous polynomials in twelve variables, one of 
degree (1,1,0,0), one of degree (0,0,1,1), and nine of degree (2, 2, 2, 2). 


We prove this result by sufficiently many random computations with 
Macaulay 2 . A slightly simplified version of the code is shown in Listing 
in Appendix 

Let us look at the result in more detail. The first two bilinear generators 
are the familiar 6 x 6-determinants 


( 6 ) 


det 


Ai 

A2 


Ul 

0 


0 

U 2 


and 


det 


Ai 

A2 


Vl 

0 


0 

V2 


These cut out two copies of the multiview threefold Va C (P^)^, in separate 
variables, for A 1—)■ u = [ui,U2) and Y eA v = [vi,V2)- If we write the two 
bilinear forms in @ as uj FU2 and vj Fv2 then T is a real 3 x 3 -matrix of 
rank 2 , known as the fundamental matrix [ 5 l § 9 ] of the camera pair [Ai, A2). 
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The rigid multiview variety V{Ja) is a divisor in x Va C (P^)^ x (P^)^. 
The nine octics that cut out this divisor can be understood as follows. We 
write B and C for the 6 x 6-matrices in Q, and Bi and C* for the matrices 
obtained by deleting their zth rows. The kernels of these 5 x 6-matrices are 
represented, via Cramer’s Rule, by AsR* and AsCj. We write AsRj and AsCj 
for the vectors given by their first four entries. As in ([^, these represent 
the two world points X and Y in P^. Their coordinates are bilinear forms 


in (mi,M 2) or {vi,V2), where each coefficient is a 3 x 3 -minor of 


Ai 

A2 


For 


instance, writing for the (j, k) entry of Ai, the first coordinate of A5R1 is 


t 32 „ 23„34 


32 „24„33 


(CLi dr) CL‘-) CL-] Ojr) Q-o Qj-\ CLr) CL’ 


^ 33 ^ 22^34 


+( 


, 32 ^ 13^34 


32 ^ 14^33 


Qi-i CJjQ CL‘-) CL-] CLi-) CLi) CL-t CLi) CL’ 


^ 33 ^ 12^34 


^ 32 ^ 13^24 


32 ^ 14^23 


(Oil di) di) d-\ di) di) d-\ di) d 


^ 33 ^ 12^24 


+( 


, 22 ^ 23^34 


22 ^ 24^33 


d-\ di) di) d-\ di) di) d-\ di) d 


., 23 ^ 22^34 


-,22^13^34 


22 ^ 14^33 


( di) di) d-\ di) di) d-\ di) d 


., 23 ^ 12^34 


+( 


, 22 ^ 13^24 


22 ^ 14^23 


, 23 ^ 12^24 


d-] di) di) d-] di) di) d-] dn d 


-,34^22^33 


-I- -I- 

+ 

+ -I- 

+ af^a 2 ‘^a 2 ^ + 
+ af^a 2 ‘^a 2 ^ + 
+ -I- 


„34„23„32', ,, ,, 

— tti 02 0,2 ) W11U2O 

„34„13„32', ,, ,, 

— O2 O2 ) O11U21 
„34„13„22'i 

— Oi 02 O2 ) U11U22 
„24„23„32'i 

— Oi O2 O2 ) U12U20 

— Oi*ol^ 02 ^) U12U21 

— af^al^ol"^) U 12 U 22 - 


Recall that the two world points in P^ are linked by a distance con¬ 
straint expressed as a biquadratic polynomial Q. We set Q{X,Y) = 
T{X,X,Y,Y), where T(», •,»,•) is a quadrilinear form. We regard T as a 
tensor of order 4 . It lives in the subspace Sym2(M^) ® Sym2(M^) — 
of ~ Here Sym^( •) denotes the space of symmetric tensors of 

order k. 

We now substitute our Cramer’s Rule formulas for X and Y into the 
quadrilinear form T. For any choice of indices l<i<j<6 and l<A:</<6, 


(7) T( AsHj, A^Bj , AsCfc , A5C1 ) 

is a multihomogeneous polynomial in {ui,U2,vi,V2) of degree (2,2, 2, 2). 
This polynomial lies in Ja but not in the ideal lA{u) + lAiv) of Va x Va, so it 
can serve as one of the nine minimal generators described in Proposition 
The number of distinct polynomials appearing in ( 7 ) equals (2) = 441 . 
A computation verifies that these polynomials span a real vector space of 
dimension 126 . The image of that vector space modulo the degree ( 2 , 2 , 2 , 2 ) 
component of the ideal lAiu) + Ia{v) has dimension 9 . 

We record three more features of the rigid multiview with n = 2 cameras. 
The hrst is the multidegree [HI § 8 . 5 ], or, equivalently, the cohomology class 
of V{Ja) in H*((P^)^,Z) ='L[ui,U2,vi,V2]/{u\,u\,v\,V2)■ It equals 

2 u\vi + 2uiU2Vi -|- 2U2V1 + 2u\v2 + 2U1U2V2 -|- 2M2n2 
-|-2miu^ -|- 2rtiui?;2 + 2?xiu| -|- 2u2v‘l + 2u2ViV2 -k 2tt2f|. 


This is found with the built-in command multidegree in Macaulay2. 

The second is the table of the Betti numbers of the minimal free resolution 
of Ja in the format of Macaulay2 [ 3 ]. In that format, the columns correspond 
to the syzygy modules, while rows denote the degrees. For n = 2 we obtain 

0 1 2 3 4 5 

total: 1 11 25 22 8 1 

0 : 1 . 

1 : . 2 . ... 

2 : . . 1 ... 

7: . 9 24 22 8 1 
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0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

total 

1 

177 

1432 

5128 

10584 

13951 

12315 

7410 

3018 

801 

126 

9 

0 

1 












1 


6 











2 


2 

21 

6 









3 



6 

36 

18 








4 



1 

12 

42 

36 

9 






5 


1 











6 


24 

108 

166 

120 

42 

6 






7 


144 

1296 

4908 

10404 

13873 

12300 

7410 

3018 

801 

126 

9 


Table 1. Betti numbers for the rigid multiview ideal with n = 3. 


The column labeled 1 lists the minimal generators from Proposition!^ Since 
the codimension of V{Ja) is 3, the table shows that Ja is not Cohen- 
Macaulay. The unique 5th syzygy has degree (3, 3,3,3) in the Z'^-grading. 

The third point is an explicit choice for the nine generators of degree 
(2, 2,2, 2) in Proposition!^ Namely, we take i = j <3 and A; = / < 3 in Q. 
The following corollary is also found by computation: 

Corollary 2. The rigid multiview ideal Ja forn = 2 is generated by Ia{u) + 
Ia{v) together with the nine polynomials Q{f\hBi, f\^Ck) for 1 < i, A; < 3. 

We next come to the case of three cameras: 

Proposition 3. For n = 3, the rigid multiview ideal Ja is minimally gener¬ 
ated by 177 polynomials in 18 variables. Its Betti table is given in Table\^ 

Proposition !^ is proved by computation. The 177 generators occur in 
eight symmetry classes of multidegrees. Their numbers in these classes are 

(110000) : 1 (220111) : 3 (220220) : 9 (211211) : 1 

(111000) : 1 (211111) : 1 (220211) : 3 (111111) : 1 

For instance, there are nine generators in degree (2, 2, 0, 2, 2, 0), arising from 
Proposition !^ for the first two cameras. Using various pairs among the 
three cameras when forming the matrices Bi,Bj,Ck and Ci in Q, we can 
construct the generators of degree classes (2, 2, 0, 2,1,1) and (2,1,1,2,1,1). 

Table !^ shows the Betti table for Ja in Macaulay2 format. The first two 
entries (6 and 2) in the 1-column refer to the eight minimal generators of 
Ia{'o) + lAiv). These are six bilinear forms, representing the three funda¬ 
mental matrices, and two trilinear forms, representing the trifocal tensor of 
the three cameras (cf. |2], [51 §15]). The entry 1 in row 5 of column 1 marks 
the unique sextic generator of Ja, which has Z®-degree (1,1,1,1,1,1). 

For the case of four cameras we obtain the following result. 

Proposition 4. For n = 4 , the rigid multiview ideal Ja is minimally gen¬ 
erated by 1176 polynomials in 24 variables. All of them are induced from 
n = 3. Up to symmetry, the degrees of the generators in the -grading are 

(11000000) : 1 (22001110) : 3 (22002200) : 9 (21102110) : 1 
(11100000) : 1 (21101110) : 1 (22002110) : 3 (11101110) : 1 
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We next give a brief explanation of how the rigid multiview ideals Ja were 
computed with Macaulay2 [3]. For the purpose of efficiency, we introduce 
projective coordinates for the image points and affine coordinates for the 
world points. We work in the corresponding polynomial ring 

Q[u,v]\Xo,Xi,X2,Yo,Yi,Y2]. 

The rigid multiview map 'll )a is thus restricted to x The prime ideal 
of its graph is generated by the following two classes of polynomials: 

(1) the 2x2 minors of the 3x2 matrices 

[ Ai-{Xo,Xi,X2,lV \ui], [ A,-{Yo,Yi,Y2Ay | ] , 

(2) the dehomogenized distance constraint 

Q((Xo,Xi,X2,i)^,(yo,Fi,y2,i)^). 

From this ideal we eliminate the six world coordinates {Xq, Xi, X 2 , Yq,Yi,Y 2 }. 

For a speed up, we exploit the group actions described in Section We 
replace A = [Ai,An) and Q = Q{X,Y) by A' = (MiAiN,MnAnN) 
and Q' = Q{N~^X, N~^Y). Here Mj G GL 3 (M) and N G GL 4 (M) are chosen 
so that Al is sparse. The modification to Q is needed since we generally 
use X ^ SE(3,M). The elimination above now computes the ideal (Mi (8) 

... G) Mn)JA, and it terminates much faster. For example, for n = 4, the 
computation took two minutes for sparse A' and more than one hour for non- 
sparse A. For re = 5, Macaulay2 ran out of memory after 18 hours of GPU 
time for non-sparse A. The complete code used in this paper can be accessed 
via http://www3.math.tu-berlin.de/combi/dmg/data/rigidMulti/, 

One last question is whether the Grobner basis property in [3l §2] extends 
to the rigid case. This does not seem to be the case in general. Only in 
Proposition!^ can we choose minimal generators that form a Grobner basis. 

Remark 5. Let re = 2. The reduced Grobner basis of Ja in the reverse 
lexicographic term order is a minimal generating set. For a generic choice 
of cameras the initial ideal equals 

in( Ja) = {U10U20, V10V20, W?0^21^10^21) ^10^21^11 ^20^21, «?0^21^11^20J 
^11^20^10^2D uI^U20U21vIqvI^, W20I^21 , 

u\]^U20U2lvliV20V21, u\iU20U21vIivIq) . 

For special cameras the exact form of the initial ideal may change. However, 
up to symmetry the degrees of the generators in the Z'^-grading stay the 
same. In general, a universal Grobner basis for the rigid multiview ideal Ja 
consists of octics of degree (2, 2, 2, 2) plus the two quadrics Q. This was 
verihed using the Gfan [6] package in Macaulay2. Analogous statements do 
not hold for re > 3. 

3. Equations for the Rigid Multiview Variety 

The computations presented in Section 2 suggest the following conjecture. 

Conjecture 6. The rigid multiview ideal Ja is minimally generated by 
|re® — |re^ + + |re^ + — ^re polynomials. These polynomials come 
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n\degree 

2 

3 

6 

7 

8 

total 

timing (s) 

2 

2 




9 

1 

< 1 

3 

6 

2 

1 

24 

144 

177 

14 

4 

12 

8 

16 

240 

900 

1176 

130 

5 

20 

20 

100 

1200 

3600 

4940 

24064 


Table 2. The known minimal generators of the rigid mul¬ 
tiview ideals, listed by total degree, for up to five cameras. 
There are no minimal generators of degrees 4 or 5. Average 
timings (in seconds), using the speed up described above, are 
in the last column. 


from two triples of cameras, and their number per class of degrees is 


( 110 .. 000 ..) : 

1 . 2 © 

( 220 .. 111 ..) 

3-2©© 

( 220 .. 220 ..) : 

tc 

( 211 .. 211 ..) 


( 111 .. 000 ..) : 

1 • 2© 

( 211 .. 111 ..) 

l- 2 »(”,-‘)© 

( 220 .. 211 ..) : 

3.2„©("-') 

( 111 .. 111 ..) 

1-©^ 


At the moment we have a computational proof only up to n = 5. Table 
offers a summary of the corresponding numbers of generators. 

Conjecture implies that V{Ja) is set-theoretically defined by the equa¬ 
tions coming from triples of cameras. It turns out that, for the set-theoretic 
description, pairs of cameras suffice. The following is our main result: 

Theorem 7. Suppose that the n focal points of A are in general position 
in The rigid multiview variety V[Ja) is cut out as a subset ofVAxVA 
by the 9 ( 2 )^ octic generators of degree class (220..220..). In other words, 
equations coming from any two pairs of cameras suffice set-theoretically. 

With notation as in the introduction, the relevant octic polynomials are 

for all possible choices of indices. Let Ha denote the ideal generated by these 
polynomials in M[ri, u], the polynomial ring in 6n variables. As before, we 
write lAiu) + lA{v) for the prime ideal that defines the 6 -dimensional variety 
Va X Va in (P^)"' X (P^)”. It is generated by 2 ( 2 ) bilinear forms and 2 ( 3 ) 
trilinear forms, corresponding to fundamental matrices and trifocal tensors. 
In light of Hilbert’s Nullstellensatz, Theorem [7] states that the radical of 
Ha + Ia{u) + Ia{v) is equal to Ja- To prove this, we need a lemma. 

A point u in the multiview variety Va C (P^)"" is triangulable if there exists 
a pair of indices (j, k) such that the matrix B^^ has rank 5. Equivalently, 
there exists a pair of cameras for which the unique world point X can be 
found by triangulation. Algebraically, this means X = A^Bj^ for some i. 

Lemma 8 . All points in Va are triangulable except for the pair of epipoles, 
(ei<_ 2 , e 2 <-i), in the case where n = 2. Here, the rigid multiview variety 
V{Ja) contains the threefolds VA{u)x{ei.i- 2 , € 2 . 1 - 1 ) and (ei<_ 2 , e 2 <-i)xI/) 4 (u). 
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Proof. Let us first consider the case of n = 2 cameras. The first cfaim hofds 
because the back-projected lines of the two camera images ui and U 2 always 
span a plane in except when ui = ei <_2 and U 2 = e 2 <-i. In that case both 
back-projected lines agree with the common baseline /3i2- Alternatively, we 
can check algebraically that the variety defined by the 5 x 5-minors of the 
matrix B consists of the single point (ei<_ 2 , e 2 <-i). 

For the second claim, fix a generic point X in and consider the surface 

(8) = {y e p3 : Q(x,y) = 0}. 

Working over C, the baseline / 3 i 2 is either tangent to or it meets that 
quadric in exactly two points. Our assumption on the genericity of X implies 
that no point in the intersection / 3 i 2 H X^ is a focal point. This gives 

(9) {A^X,A2X,A^Yx,A2Yx) = (AiX,A 2 X, 61 ^ 2 , 62 ^ 1 ). 

The point (AiX, A 2 X) lies in the multiview variety Va{u). Each generic 
point in Va(u) has this form for some X. Hence Q proves the desired 
inclusion Va(u) x (ei^ 2 , 62 ^ 1 ) C V(Ja)- The other inclusion (ei<_ 2 , e 2 <-i) x 
Va(v) C V{Ja) follows by switching the roles of u and v. 

If there are more than two cameras then for each world point X, due to 
general position of the cameras, there is a pair of cameras such that X avoids 
the pair’s baseline. This shows that each point is triangulable if n > 3. □ 

Proof of Theorem It follows immediately from the definition of the ideals 
in question that the following inclusion of varieties holds in (P^)*^ x (P^)*^: 

V{Ja) C V{Ia{u) + Ia{v) + Ha). 

We prove the reverse inclusion. Let (u, v) be a point in the right hand side. 

Suppose that u and v are both triangulable. Then u has a unique preimage 
X in P^, determined by a single camera pair Likewise, v has a 

unique preimage y in P^, also determined by a single camera pair {Aj 2 , Ak^}- 
There exist indices ii,i 2 £ {1, 2, 3,4, 5, 6 } such that 

X = and Y = l{^cif\ 

Suppose that (u, u) is not in V{Ja). Then Q{X,Y) / 0. This implies 

Q{X,Y) = T{X,X,Y,Y) = ^(A5^I^"^A5H/^"^A5C'g"^A5Cg"^) / 0, 

and hence {u,v) 0 V{Ha). This is a contradiction to our choice of {u,v). 

It remains to consider the case where v is not triangulable. By Lemma 
we have n = 2, as well as u = (ei<_ 2 , e 2 <-i) and {u,v) G V{Ja)- The case 
where u is not triangulable is symmetric, and this proves the theorem. □ 

The equations in Theoremare fairly robust, in the sense that they work 
as well for many special position scenarios. However, when the cameras 
Ai, A 2 ,..., An are generic then the number 9 ( 2 )^ of octics that cut out the 
divisor V{Ja) inside Va x Va can be reduced dramatically, namely to 16. 

Corollary 9. As a subset of the 6-dimensional ambient space Va x Va, the 
6-dimensional rigid multiview variety V{Ja) is cut out by 16 polynomials of 
degree class (220..220..). One choice of such polynomials is given by 

g(A5Hi2,A5Cf), g(A5Hp,A5Cf) 

Q(A5Hf,A5Cf), g(A5H/3,A5Cf) 


for all I < i,k < 2. 
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Proof. First we claim that for each triangulable point u at least one of the 
matrices or has rank 5, and the same for v with or . We 
prove this by contradiction. By symmetry between u and v, we can assume 
that rk(B^^) = rk(B^^) = 4. Then = e 3 <_i, U 2 = e 2 <-i, and ui = ei <_2 = 
ei<_ 3 . However, this last equality of the two epipoles is a contradiction to the 
hypothesis that the focal points of the cameras Ai,A 2 ,A^ are not collinear. 

Next we claim that if B^^ has rank 5 then at least one of the submatrices 
B\^ or B^ has rank 5, and the same for and Note that the 

bottom 4x6 submatrix of has rank 4, since the first four columns are 
linearly independent, by genericity of Ai and A 2 . The claim follows. □ 


4. Other Constraints, More Points, and No Labels 


In this section we discuss several extensions of our results. A first obser¬ 
vation is that there was nothing special about the constraint Q in Q. For 
instance, fix positive integers d and e, and let Q(A, T) be any irreducible 
polynomial that is bihomogeneous of degree (d, e). Its variety V{Q) is a 
hypersurface of degree (d, e) in x P^. The following analogue to Theorem 
holds, if we define the map if a as in ([^. 

Theorem 10. The closure of the image of the map if a is cut out inVAxVA 
by 9 ( 2 )^ polynomials of degree class (d, d, 0,..., e, e, 0,...). In other words, 
the equations coming from any two pairs of cameras suffice set-theoretically. 


Proof. The tensor T that represents Q now lives in Sym^(]R‘^) 0 Symg(M^). 
The polynomial Q vanishes on the image of if a and has degree (d, d, e, e). 
The proof of Theorem remains valid. The surface in ([^ is irreducible 
of degree e in P^. These polynomials cut out that image inside VaxVa. D 

Remark 11. In the generic case, we can replace 9 ( 2 )^ by 16, as in Corollary 

Another natural generalization is to consider m world points Xi ,..., X^n 
that are linked by one or several constraints in (P^)™. Taking images with 
n cameras, we obtain a variety V{Ja) which lives in instance, 

if m = 4 and Ai, A 2 ,A 3 , A 4 are constrained to lie on a plane in P^, then 
Q = det(Ai, A 2 , A 3 , A 4 ) and V{Ja) is a variety of dimension 11 in (P^)*^*^. 
Taking 6 x 6 -matrices B, C, D, E as in Q for the four points, we then form 

( 10 ) det{A5Bi,7\5Cj,A5Dk,7\5Ei) for all 1 < i,j,k,l < 6 . 

For n = 2 we verified with Macaulay2 that the prime ideal Ja is generated 
by 16 of these determinants, along with the four bilinear forms for Va^- 


Proposition 12 . The variety V{Ja) is cut out in Va^ by the 16 ( 2 )^ poly¬ 
nomials from (10). In other words, the equations coming from any two pairs 
of cameras suffice set-theoretically. 


Proof. Each polynomial (10) is in Ja. The proof of Theoremj^remains valid. 
The planes {Xi, Xj, Xk)^ intersect the baseline / 3 i 2 in one point each. □ 


To continue the theme of rigidity, we may impose distance constraints on 
pairs of points. Fixing a nonzero distance dij between points i and j gives 

Q,, = ( AoA, 3 - A.oWa)" + (Wi A,3 - A,i A3)" + (A2A,3 - A,2 As)" - 
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We are interested in the image of the variety V = V{Qij : 1 < i < j < m) 
under the multiview map iI^a that takes to instance, for 

m = 3, we consider the variety V = V{Qi 2 , Qis, <523) in (P^)^, and we seek 
the equations for its image under the multiview map ifjA into (P^)^”. Note 
that V has dimension 6, unless we are in the collinear case. Algebraically, 

(11) (dl2 + dl3+I^23)(c^l2 + dl3—d23)(c^l2—dl3 + d23)( —<^ 12 +^ 13 +^ 23 ) = 0. 
If this holds then dim(V) = 5. The same argument as in Theorem yields: 

Corollary 13. The rigid multiview variety V'a(V) dimension six, unless 
(H holds, in which case the dimension is five. It has real points if and only 
if di 2 ,di 3 ,d 23 satisfy the triangle inequality. It is cut out in Va^ by 27 ( 2 )^ 

biquadratic equations, coming from the 9 ( 2 )^ equations for any two of the 
three points. 

In many computer vision applications, the m world points and their im¬ 
ages in P^ will be unlabeled. To study such questions, we propose to work 
with the unlabeled rigid multiview variety. This is the image of the rigid 
multiview variety under the quotient map ((P^)™)*^ —)• (Sym^ (P2))". 

Indeed, while labeled configurations in the plane are points in (P^)”^, 
unlabeled configurations are points in the Chow variety Sym^(P^). This 
is the variety of ternary forms that are products of m linear forms (cf. [TJ 
§8.6]). It is embedded in the space P^ 2 )“^ of all ternary forms of degree m. 

Example 14. Let m = n = 2. The Chow variety Sym 2 (P^) is the hyper¬ 
surface in P^ defined by the determinant of a symmetric 3 x 3-matrix {aij). 
The quotient map (P^)^ —)• Sym 2 (P^) C P^ is given by the formulas 

aoo = 2 mioUio, an = 2uiiuii, 022 = 2ui2Ui2, 

001=11111110 + ^10^11, ao2 = ai2Uio + U101112, 012 = U121111 +'0111112- 
Similarly, for the two unlabeled images under the second camera we use 

boo = 2tt2oii20, bn = 2m2iii21, ^22 = 2 U 22 V 22 , 

boi = II21II20 + U20V2I, bo2 = 0221120 + O20II22, bl2 = U22V2I + 0211122- 

The unlabeled rigid multiview variety is the image oi V{Ja) C Va x Va 
under the quotient map that takes two copies of (P^)^ to two copies of 
Sym 2 (P)^ C P^. This quotient map is given by {ui,vi) 1 —)• a, (02,112) b. 

We first compute the image of Va x Va in P® x P^, denoted Sym 2 (VA). Its 
ideal has seven minimal generators, three of degree (1,1), and one each in 
degrees (3,0), (2,1), (1, 2), (0, 3). The generators in degrees (3, 0) and (0, 3) 
are det(aij) and det{bij). The five others depend on the cameras Ai, A 2 . 

Now, to get equations for the unlabeled rigid multiview variety, we in¬ 
tersect the ideal Ja with the subring M[a, 6] of bisymmetric homogeneous 
polynomials in M[u,u]. This results in nine new generators which represent 
the distance constraint. One of them is a quartic of degree (2,2) in {a,b). 
The other eight are quintics, four of degree (2, 3) and four of degree (3, 2). 

Independently of the specific constraints considered in this paper, it is of 
interest to characterize the pictures of m unlabeled points using n cameras. 
This gives rise to the unlabeled multiview variety Sym,„(I/ 4 ) in (Pi 2 j . 
It would be desirable to know the prime ideal of Sym^(I/ 4 ) for any n and m. 



RIGID MULTIVIEW VARIETIES 
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Appendix A. Computations 

We performed several random experiments in this paper. Our hardware 
was a cluster with Intel Xeon X2630v2 Hexa-Cores (2.8 GHz) and 64GB 
main memory per node. The software was Macaulay2, version 1.8.2.1 [Ij. 
All computations were single-threaded. 

The tests were repeated several times with random input. The exact 
running times vary, even with identical input; the Table lists the average 
values. It is not surprising that increasing n, the number of cameras, in¬ 
creases the running times considerably. Therefore we adapted the number 
of experiments according to n. 

For all the statements in Section 2 regarding two cameras, (i.e. n = 2) 
we performed at least 1000 computations, with one exception. The state¬ 
ment regarding the universal Grobner basis in Remark is based on 20 
experiments. Regarding three and four cameras (i.e., n G {3,4}) we per¬ 
formed at least 100 computations each. For n = 5 we performed at least 
20 computations each. Example was checked with 50 choices of random 
cameras. 

In Listing!^ we show Macaulay2 code which can be employed to establish 
Proposition The complete code for all our results can be accessed via 
http://www3.math.tu-berlin.de/combi/dmg/data/rigidMulti/, 

Lines 1-4 define the rings in which the computations take place. Lines 
6-7 produce random camera matrices. Here the code shown differs slightly 
from the code used. What we omitted is the extra code which checks the 
matrices before they are processed. To assert general position we check that 
none of the minors vanish as in [3l §2]. However, our experiments suggest 
that it suffices to check that the focal points of the cameras are in linear 
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general position. The multiview map cjiA from Q is encoded in lines 11-14. 
Line 15 is the rigid constraint Q. The actual computation is the elimination 
in line 16. The rigid multiview ideal Ja is defined in lines 17-18, and the 
final output are the multidegrees of Ja- 

1 
2 

3 

4 

5 

6 

7 

8 
9 

10 
11 
12 

13 

14 

15 

16 

17 

18 

19 

20 
21 


In these computations the world coordinates are dehomogenized by setting 
the last coordinate to 1, as explained at the end of Section Notice that 
the code below line 7 does not need to be modified if we increase n. 


R1 = QQ[u_(l,0) ..u_(l,2)] ** QQ[u_(2,0) ..u_(2,2)] ** 

QQ[v_(l,0) ..v_(l,2)] ** QQ[v_(2,0) ..v_(2,2)] ; 

R2 = QQ[X_0..X_2] ** QQ[Y_0. .Y_2] ; 

S = R1 ** R2; 

n = 2; 

for i from 1 to n do (A_i = random (ZZ“3,ZZ“4, Height=>20) ;); 

I = ideal 0; 

for j from 1 to n do ( 

1 = 1+ minors(2,A_j * (genericMatrix(S ,X_0,3,1) I ImatrixllUl) I 
genericMatrixCS ,u_(j,0),3,1) ) ; 

1 = 1+ minors(2,A_j * (genericMatrixCS ,Y_0,3,1) I ImatrixllUl) I 
genericMatrixCS ,v_(j,0),3,1)) ; ) ; 

1=1+ ideal((X_0-Y_0)*2 + (X_l-Y_l)“2 + (X_2-Y_2)‘2-1); 

I = eliminate({X_0,X_l,X_2,Y_0,Y_l,Y_2},I) ; 

F = mapCRl ,S); 

J = FCI); 

degrees (J) 

Listing 1. Compute Ja for two cameras 
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